Enhanced input using flashing electromagnetic radiation

ABSTRACT

Enhanced input using flashing electromagnetic radiation, in which first and second images of an object are accessed. The first image being captured while the object is illuminated with projected electromagnetic radiation, and the second image being captured while the projected electromagnetic radiation is extinguished. A position of the object relative to the screen based on comparing the first and second images is determined. An application is controlled based on the determined position.

CROSS-REFERENCES TO RELATED APPLICATIONS

The present application is a continuation patent application of U.S.patent application Ser. No. 12/032,496, filed Feb. 15, 2008, entitled“ENHANCED INPUT USING FLASHING ELECTROMAGNETIC RADIATION,” now U.S. Pat.No. 8,116,518, which claims the benefit of U.S. Provisional ApplicationNo. 60/890,108, filed Feb. 15, 2007, both of which are incorporatedherein by reference in their entirety.

TECHNICAL FIELD

The present disclosure generally relates to user input.

BACKGROUND OF THE INVENTION

If an object in the vicinity of a screen is illuminated by both aconstant source of light and ambient light, the ambient light may not besufficiently distinguishable from the constant source of light to allowa reflection of the constant source off of the object to be used foruser input.

BRIEF SUMMARY OF THE INVENTION

In one general implementation, an object is illuminated with flashinglight, where ambient light behind a screen is detected during anoff-period of the flashing light and filtered from an image of thescreen, in real-time or near-real-time. Based on the filteringoperation, a two-dimensional or three-dimensional position or motion ofthe object relative to the screen may be discerned and used to controlan application, such as by mapping the detected position or motion to acontrol input or mouse event.

In a further general implementation, first and second images of anobject are accessed. The first image being captured while the object isilluminated with projected electromagnetic radiation, and the secondimage being captured while the projected electromagnetic radiation isextinguished. A position of the object relative to the screen based oncomparing the first and second images is determined. An application iscontrolled based on the determined position.

Implementations may include one or more of the following features.Electromagnetic radiation may be projected as a first light curtainalong and parallel to the second side of the screen. Electromagneticradiation may be projected as a second light curtain adjacent andparallel to the second side of the screen, the first light curtain andthe second light curtain being parallel to each other and disposed at afirst location and a second location with respect to the second side ofthe screen. The object may include a portion of a body of a person. Asecond position may be determined and the application may be controlledbased on the determined position and the second position. Determining aposition of the object relative to the screen based on comparing thefirst and second images may include subtracting the first and secondimages to create a difference image. Determining a position of theobject relative to the screen may include determining the location ofthe object in the difference image.

In some example implementations, controlling an application based on thedetermined position may include controlling a cursor. The projectedelectromagnetic radiation may include infrared radiation. A visibleimage may be projected onto the screen, the object may be in motion, andcontrolling an application based on the determined position may includecontrolling a cursor on the visible image such that motion of the cursorcorresponds to the motion of the object. The motion of the object mayinclude a gesture made by a person.

In some example implementations, accessing first and second images mayinclude accessing first and second images of an object and a secondobject, the object and the second object disposed on the same side ofthe screen, and a position of a second object relative to the screen maybe determined. A shadow cast on the screen may be modeled, where theshadow may be cast by the object while the object is illuminated withthe projected electromagnetic radiation. It may be determined, based onthe model, that the shadow obscures a second object. Determining aposition of the second object relative to the screen may includeaccounting for the shadow.

In another general implementation, a system includes a translucentscreen including a first side and a second side and a source ofelectromagnetic radiation, the source configured to alternate betweenemitting the electromagnetic radiation and extinguishing theelectromagnetic radiation at a rate of flashing. A sensor is positionedon a first side of the screen and configured to sense electromagneticradiation at the rate of flashing, and the electromagnetic radiation isreflected from an object disposed in the vicinity of the screen on thesecond side of the screen. The system also includes a processorconfigured to determine a position of the object based on the sensedelectromagnetic radiation, and control an application based on thedetermined position of the object.

Implementations may include one or more of the following features. Thescreen may include edges, and the source of electromagnetic radiationmay include a linear array of emitters positioned along one or more ofthe edges of the screen such that electromagnetic radiation is projectedon the, and parallel to, the second side of the screen. The screen maybe a parallelogram. The source of electromagnetic radiation may includea series of linear arrays of emitters, and each of the linear arrays maybe positioned on the second side of the screen at a distance from asurface of the screen. The source of electromagnetic energy may emitdiverging electromagnetic energy, a channel may be adjacent to thesource, where the channel may be configured to collimate the emittedelectromagnetic energy. The channel may include a block of material witha passageway extending through the block from an entry to an exit, anddiverging electromagnetic radiation may impinging on the entry and aportion of the diverging electromagnetic radiation may emerge from theexit as collimated electromagnetic radiation. The channel may include alens. The source of electromagnetic radiation may include an emitterpositioned on the first side of the screen and the electromagneticradiation passes through the screen and reflects off the object. Thesource of electromagnetic radiation may be a light emitting diode. Thelight emitting diode may emit infrared radiation. The sensor may be anarray of pixels. The processor also may be configured to model a shadowcast by the object and determine a position of a second object disposedon the second side of the screen based on the modeled shadow.

Implementations of any of the techniques described above may include amethod or process, a system, or instructions stored on a storage device.The details of particular implementations are set forth in theaccompanying drawings and description below. Other features will beapparent from the following description, including the drawings, and theclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A to 1C provide a contextual diagram of an exemplary system.

FIGS. 2 to 4, 7A to 7D, 9, and 10 show exemplary systems that allowsusers to interact with an application.

FIG. 5 is a flowchart illustrating an exemplary process for determininga position of an object.

FIG. 6 illustrates two example input images and a resulting differenceimage.

FIGS. 8A and 8B illustrate an exemplary source of electromagneticradiation.

FIG. 11 shows an exemplary system and concomitant controlledapplication.

DETAILED DESCRIPTION OF THE INVENTION

In one general implementation, an object is illuminated with flashinglight, where ambient light behind a screen is detected during anoff-period of the flashing light and filtered from an image of thescreen, in real-time or near-real-time. Based on the filteringoperation, a two-dimensional or three-dimensional position or motion ofthe object relative to the screen may be discerned and used to controlan application, such as by mapping the detected position or motion to acontrol input or mouse event.

For instance, and referring to FIG. 1A, a system 100 allows users 102,103, and 104 to interact with an application 105 that is displayed on atranslucent screen 110 without necessarily touching the screen 110. Inthe example shown, the users 102, 103, and 104 move objects through acurtain of light (which also may be referred to as a “light curtain”and/or a “plane of light”) projected parallel to and in front of thescreen 110 in order to interact with the application 105. In the exampleshown in FIG. 1A, the application 105 is a “Grab the Plant” game.

The light curtain is formed by electromagnetic radiation emitted fromsources 115 and 120. The electromagnetic radiation may be light. Tointeract with the application 105, the users 102, 103, and 104 may moveor place objects, such as a hand or a stick, towards the screen 110 tobreak the light curtain. Movement, hovering, and/or placement of objectsin the light curtain may cause the system 100 to issue a sequence ofmouse events (or other I/O events) to the application 105, effecting thecontrol of the application 105. Thus, the users 102, 103, and 104 cancontrol the application 105 by moving, placing, or hovering objects suchthat the objects are illuminated by the light curtain.

As described in more detail below, motion, placement, and/or position ofthe objects with respect to the screen 110 is determined and used tocontrol the application 105. In particular, objects placed in the lightcurtain reflect electromagnetic energy projected from the sources 115and 120, and the reflected electromagnetic energy is sensed by a sensorsuch as a pixel, an array of pixels, or a camera. Positions or motion ofthe objects are determined based on the sensed energy, and theapplication 105 is controlled based on the determined position ormotion. In addition to the objects reflecting light from the sources 115and 120 on to the screen 110, the objects also may reflect ambientlight, or the ambient light may itself illuminate the screen 110. Theambient light may appear in images and would otherwise be erroneouslydetected as an object.

In order to distinguish light from the sources 115 and 120 that isreflected by the objects from ambient light spots that may be mistakenas objects, the sources 115 and 120 emit electromagnetic radiation in apredetermined pattern or sequence (e.g., the sources 115 and 120 and thelight curtain are flashing) such that objects illuminated by the lightcurtain also are illuminated at the predetermined pattern or sequence.Thus, actual objects appear in images collected while the sources 115and 120 emit radiation, whereas ambient light spots appear in allimages, including those collected while the sources 115 and 120 are notemitting electromagnetic radiation. By comparing images collected whilethe sources 115 and 120 emit electromagnetic radiation to imagescollected while the sources 115 and 120 are extinguishingelectromagnetic energy, actual objects may be distinguished from strayspots on the screen 110 that are caused by ambient light. The sensor issynchronized, or tuned, with the predetermined pattern or sequence suchthat the sensor alternates collecting images of the screen while theobjects are illuminated by the sources 115 and 120 and while the objectsare not illuminated by the sources 115 and 120.

In the example shown in FIG. 1A, the “Grab the Plant” game 105 allowsusers to select a plant from among plants 121, 122, 123, and 124 thatare projected on the screen 110. Once a plant is selected, the game 105presents an enlarged and shaded version of the selected plant. The user102 selects the plant 121 with his right hand 134, and the user 104selects the plants 123 and 124. In this example, the user 102 placeshands 132 and 134 into the light curtain and selects the plant 121 by,for example, making a gesture with the hand 134 while the hand 134 isilluminated by the light curtain or by hovering the hand over the plant121 while the hand 134 is illuminated by the light curtain. The hands132 and 134 cast shadows 135 and 137, respectively. The user 104 placeshands 141 and 143 in the light curtain and selects the plants 123 and124. The hands 141 and 143 cast shadows 145 and 147, respectively.Rather than selecting a particular plant with a hand or other object,the user 103 has placed a large portion 149 of their body in the lightcurtain. The body of the user 103 may or may not cast a shadow 150,depending upon the configuration of the light curtain.

Referring to FIG. 1B, representations of positions of the hands 132,134, 141, 143, and the portion 149 of the body that breaks the lightcurtain are shown. Positions 132 b and 134 b are determined tocorrespond to the positions of the hands 132 and 134 with respect to thescreen 110, and positions 141 b and 143 b correspond to the positions ofthe hands 141 and 143 with respect to the screen 110. A position 160corresponds to the position of the portion 149 of the body of the user103. As discussed in more detail below, the system 100 analyzes the sizeand shape, among other features, of reflections on the screen 110 madeby objects illuminated by the curtain of light, to determine how tocontrol the application 105. In this example, the portion 149 of thebody of the user 103 is too large to be a selection, and thus theposition 160 may be deemed an error and ignored.

Referring to FIG. 1C, the application 105 is controlled based on thedetermined positions. The position 134 b corresponds to a selection ofthe plant 121, and the plant 121 is enlarged into the plant 126 as aresult of the selection. Similarly, the position 141 b and the position143 b correspond to selections of the plants 123 and 124, respectively,and the plants 123 and 124 are enlarged into the plants 128 and 129,respectively. In contrast, the error 160 does not select the plant 122,as it is automatically determined to too large to be a control selectionor because it does not touch a region of a screen upon which a plant isprojected. The position 132 b does not correspond to a plant, thus theposition 132 b does not cause any of the plants 121-124 to becomeenlarged.

Referring to FIG. 2, a side view of a system 200 is illustrated. Thesystem 200 allows a user to interact with an application, which may berunning on a computing device 201. The application is projected by aprojector 202 onto a translucent screen 205. The application may be anapplication such as the game “Grab the Plant” 105 discussed above withrespect to FIGS. 1A-1C. In other examples, the application may be anyother computer program such as, for example, a presentation application,image and/or video editing software, or an electronic mappingapplication.

The system 200 includes the projector 202, the screen 205, a source 210that projects electromagnetic radiation into a light curtain 215, and asensor 220 that images the screen 205. In particular, the sensor 220senses electromagnetic radiation reflected from an object 230 onto thescreen 205 as the object is illuminated by the light curtain 215. Theradiation reflected from the object 230 causes a bright spot 232 toappear on the screen 205. The sensor 220 also images other bright spotson the screen, such as bright spots caused by an ambient light source235. In the example shown, the bright spot 237 is caused by an ambientlight source 235. In addition to reflecting electromagnetic radiationemitted from the source 210, the object 230 also may reflect ambientlight from an ambient light source 235. The source 210 may be includedin a frame 240 that fits around the screen 205.

The screen 205 is a translucent, transparent, or semi-transparent screenonto which the projector 202 projects an image of the application. Thescreen 205 may be made from, for example, glass, plexiglass,semi-transparent plastic, or any other transparent or semi-transparentmaterial. For instance, the screen may be a window in a store, or aspecialized screen (for example in a crowded area such as a mall orairport) upon which images are projected and interaction is solicited,or the screen may be disposed in an entertainment facility such as atheater, arcade, or bar. The projector 202 may include a filter 203 toremove infrared radiation from the image projected by the projector 202.

The source 210 may be any emitter of electromagnetic radiation. Forexample, the source 210 may be a laser, a light emitting diode (LED), ora broadband light source. The source 210 alternates between emittingelectromagnetic radiation and not emitting electromagnetic radiation.Thus, the source may be considered to flash at a flashing sequence orpattern, and the light curtain 215 is a flashing light curtain incontrast to a light curtain that provides constant illumination. In someimplementations, the electromagnetic radiation emitted from the source210 is infrared radiation including wavelengths beyond the perception ofa human typical eye. Thus, in these implementations, the light curtain215 is invisible to a user of the system 200. Similarly, in theseimplementations, light reflected by the object 230 onto the screen 205is also invisible to the user. Thus, the light reflected onto the screen205 does not interfere with the user's view of the projectedapplication. In other implementations, the source 210 may emitelectromagnetic radiation with wavelengths in the visible band (e.g.,400 nanometers to 700 nanometers).

The sensor 220 is synchronized with the source 210 such that the sensor220 alternates between sensing data while the source 210 is emittingelectromagnetic radiation and while the source 210 is extinguishing theelectromagnetic radiation at a rate of flashing. For example, the sensor220 may be a camera that captures frames, or images, at thirty, sixty,or one hundred twenty frames per second. In this example, the source 210emits electromagnetic energy such that every other frame collected bythe source 210 is an image captured while the light curtain 215 isilluminated. A filter 223 may be positioned in front of and/orintegrated with the sensor 220 such that electromagnetic radiation thathas a wavelength of the radiation emitted from the source 210 is sensedby the sensor 220.

The object 230 may be a finger or a hand of a user, a pointer, a stylus,or another similarly sized object. The object 230 reflectselectromagnetic radiation on to the screen 205 as the object 230 isilluminated by the light curtain 215. The reflected electromagneticradiation passes through the screen 205 and is sensed by the sensor 220.The ambient light source 235 may be an incandescent light bulb (such asfrom a canned ceiling fixture), or sunlight. Radiation from the ambientlight source 235 also appears on the screen as bright spots, such as thebright spot 237. The bright spot 237 on the screen caused by the ambientlight source 235 is also sensed by the sensor 220 and is present in thedata collected by the sensor 220. The ambient light source 235 may bestationary or, at most, only slowly moving. Thus, the ambient lightsource 235 causes a bright spot 237 to appear in the data collected bythe sensor 220 over a particular time period, and the bright spot 237appears in generally the same portion of the data collected at differenttimes within the time period. Subtracting, or otherwise comparing thedata over the time period collected while the source 210 emits radiationto data collected while the source is not emitting radiation results inthe removal of the energy from the ambient light source 235 from thedata. However, because the electromagnetic radiation reflected from theobject 230 as a result of the object being illuminated by the lightcurtain 215 is only present in data collected by the sensor 220 whilethe source 210 is emitting electromagnetic radiation, comparing the datadoes not result in the removal of an image of the object 230. This, useof the flashing light curtain 215 allows bright spots on the screen thatare caused by the presence of actual objects (such as the bright spot232) to be distinguished from spots on the screen 205 that are caused byradiation from the ambient light source 235 (such as the bright spot237).

In the implementation shown in FIG. 2, the sensor 220 is located on afirst side 240 of the screen 205 and the source 210 is located on asecond side 245 of the screen 205. Thus, the light curtain 215 is alsolocated on the second side 245 of the screen 205. In implementations inwhich the sensor 220 and the source 210 are located on opposite sides ofthe screen 205, electromagnetic radiation reflected from the object 230passes through the screen 205 and is sensed by the sensor 220. In otherimplementations, such as the implementation shown in FIG. 9, the source210 and the sensor 205 may both be located on the first side 240 of thescreen 205. In these implementations, the electromagnetic radiationemitted from the source 210 passes through the screen 205 andilluminates the object 230. The object 230 reflects the electromagneticradiation, and the reflected electromagnetic radiation passes throughthe screen 205 and is sensed by the sensor 220.

Referring to FIG. 3, a front view of a system 300 is illustrated. Thesystem 300 may be similar to the system 200 discussed with respect toFIG. 2. The system 300 includes a screen 305 and a source 310 thatprojects a light curtain of electromagnetic radiation parallel to and infront of the screen 305. The source 310 is configured to alternatebetween emitting the electromagnetic radiation and extinguishing theelectromagnetic radiation at a rate of flashing. In the example shown inFIG. 3, the source 310 is a linear array of emitters 311 positionedalong an edge 306 a of the screen 305. The emitters 311 may be, forexample, line lasers and/or infrared LEDs. The electromagnetic energyemitted from the emitters 311 may be divergent, and the divergentelectromagnetic energy is passed through a channel 312 to make theelectromagnetic radiation into a thin light curtain that is parallel toand above a surface of the screen 205. Details of the channel 312 arediscussed below.

The screen 305 also includes edges 306 b, 306 c, and 306 d. In otherexamples, the screen 305 may include more or fewer edges. In otherexamples, such as the example illustrated in FIG. 7, more than one edgeof the screen 305 may have a corresponding source positioned along theedge. The electromagnetic radiation projected as the light curtainilluminates objects 330 and 332, and the objects 330 and 332 castshadows 331 and 333, respectively. The objects 330 and 332 may be, forexample, hands, wands, and/or pointers. The shadow 331 partiallyobscures the object 332. Because the position of the object 330 isdetermined, and the location of the source 310 is known, the shape,size, and/or location of the shadow 331 may be modeled. Modeling theshadow 331 may be used to, for example, determine blind spots on thescreen 305 where the position of additional objects, such as the object332, may not be detected accurately or may not be detected at all.

Referring to FIG. 4, a system 400 for determining and/or trackingpositions of objects with respect to a screen is illustrated. The systemincludes a source system 410, a detection system 430, and a trackingsystem 440. The source system 410 produces a light curtain parallel toand adjacent to a surface of the screen, such as the light curtain 215discussed above with respect to FIG. 2. The detection system 430 sensesradiation reflected from an object illuminated by the light curtain, andthe tracking system 440 determines a position or motion of the objectwith respect to the screen.

The source system 410 generates a flashing light curtain or otherprojected electromagnetic illumination that may be used to illuminateobjects on one side of a screen. The source system 410 includes anemitter 412 and electronics and controls 416 to control the emitter 412.The source system 410 also may include a channel 414 to create a thinlight curtain from an emitter that emits divergent electromagneticradiation such as an LED. The electronics and controls 416 includeelectronics coupled to the emitter 412 that cause the emitter 412 toflash. For example, the electronics and controls 416 cause the emitter412 to alternate between emitting the electromagnetic radiation andextinguishing the electromagnetic radiation at a rate of flashing. Theemitter 412 may be an infrared emitter that emits infraredelectromagnetic radiation having a wavelength above 680 nanometers. Insome implementations, the emitter 412 may emit visible light.

Electromagnetic radiation 420 from the source system 410 illuminates anobject on one side of a screen, and electromagnetic radiation isreflected from the object onto the screen and sensed by the detectionsystem 430. In particular, the detection system 430 images the screen.The detection system 430 includes a sensor 432, a filter 434, sensorelectronics and controls 436, and an input/output (“I/O”) interface 438.In some implementations, the detection system 430 may be a digitalcamera, and the sensor 432 may be an array of pixels included in thecamera. The filter 434 may be a filter that blocks electromagneticradiation that has wavelengths other than the wavelengths of theelectromagnetic radiation produced by the emitter 412. The I/O interface438 may communicate with the tracking system 440 and/or the sourcesystem 410 to send a signal to the source system 410 that allows thesource system 410 and the detection system 430 to be synchronized. Forexample, in implementations where the detection system 430 is a camerathat collects data in frames, the I/O interface 438 may send a signal tothe source system 410 at the beginning of each frame. The camera maycollect thirty frames of data per second. The electronics and controls416 of the source system 410 receive the signal and control the emitter412 such that the emitter 412 alternates between emitting theelectromagnetic radiation and extinguishing the electromagneticradiation at a rate that allows every other frame of data collected bythe camera to be an image of the screen captured while the emitter 412is emitting electromagnetic radiation.

The system 400 also includes the tracking system 440. The trackingsystem 440 determines a position of an object illuminated by the lightcurtain from an image of the screen collected by the detection system430. The tracking system 440 uses the determined position to control anapplication, such the application 105 discussed above with respect toFIGS. 1A-1C. The tracking system 440 includes an analysis module 450, aninterface to an application to control 460, an input/output module 470,a processor 480, and a memory 490.

The analysis module 450 determines a position of the object illuminatedby the tracking system light curtain. The analysis module 450 includesan input routine 452 that imports data from the detection system 430into the analysis module 450. The import routine 452 also may transformthe data from the detection system 430 into a format that can beprocessed by the analysis module 450. The analysis module 450 alsoincludes a comparison module 454 that compares data collected atdifferent times by the detection module 430. For example, the comparisonmodule 454 may subtract data collected by the detection module 430 whilethe emitter is emitting radiation from data collected by the detectionmodule 430 while the emitter 412 is extinguishing radiation.

The analysis module 450 also includes a shadow modeler 456 that modelsshadows cast by the objects illuminated by the light curtain in order todetermine blind spots on the screen in which positions of objects arenot detected, accurately or at all, even when objects are present. Theanalysis module 450 also includes a position determination module 458that determines a position of an object illuminated by the light curtainwith respect to a screen. In particular, the object illuminated by thelight curtain reflects light onto the screen. As a result, an image ofthe screen captured by the detection system 430 has a bright spot at alocation corresponding to the location of the object with respect to thescreen. In some implementations, the object moves and the positiondetermination module 458 tracks the object as it moves through the lightcurtain. In some implementations, the motion of the object may be agesture of a user's hand. In these implementations, the positiondetermination module 458 determines that a particular gesture has beenmade. The particular gesture may signify that an application controlledthrough the motion of the object is to perform a certain action.

The tracking system 440 also includes an interface 460 that passes theposition determined by the position determination module 458 to anapplication that is controlled based on the determined position. Forexample, the application may be a game, and the game may be controlledbased on motion of a cursor. The motion of the cursor may correspond tothe motion of the object illuminated by the light curtain. The trackingsystem 440 also includes an input/output module 470. The input/outputmodule 470 may include a port that interfaces with the detection system430. The input/output module 470 may include a mouse, stylus, keyboard,printer, and/or any other input/output device that allows a user tointeract with the tracking system 440. The input/output module 470 mayinclude a port or interface that enables an automated process or machineto interact with the tracking system 440.

The tracking system 440 also includes a processor 480 and a memory 490.The tracking system 440 may include more than one processor and morethan one memory. The processor 480 may be a processor suitable for theexecution of a computer program such as a general or special purposemicroprocessor, and any one or more processors of any kind of digitalcomputer. Generally, a processor receives instructions and data from aread-only memory or a random access memory or both. The processor 480receives instruction and data from the components of the workflownavigation system tracking system 440 to, for example, determine theposition of objects with respect to a screen.

Referring to FIG. 5, an example process 500 for determining the positionof an object illuminated by a flashing light curtain is illustrated. Theprocess 500 may be performed by, for example, one or more processorsincluded in a system such as the systems 200, 300, and/or 400 discussedabove. For example, the process 500 may be performed by the processor480 included in the tracking system 440 discussed with respect to FIG.4.

First and second images of an object and ambient radiation are accessed(510). The first and second images are captured on a first side of ascreen and the object and the ambient radiation are disposed on a secondside of the screen. The first image is captured while the object isilluminated with projected radiation, and the second image is capturedwhile the projected radiation is extinguished. Returning briefly to FIG.2 as an example, the first side of the screen may be the first side 240of the screen 205 and the second side of the screen may be the secondside 245 of the screen 205. The object may be an object such as theobject 230 and the ambient radiation may be a source such as the ambientlight source 235. The projected radiation may be a light curtain thatflashes such as the light curtain 215.

Returning to FIG. 5, a position of the object relative to the screen isdetermined based on comparing the first and second images (520). Forexample, the first and second images may be compared by subtracting thefirst and second images. An application is controlled based on thedetermined position (530).

Referring to FIG. 6, an illustration 600 of example data used todetermine a position of an object is shown. In particular, the datashown in the illustration 600 shows the removal of ambient light. Theexample data in the illustration 600 may be used in a process, such asthe process 500, to determine the position of an object with respect toa screen in the presence of ambient radiation. A light curtain, isprojected parallel to and along one side of the screen. While the object(in this example, the object is a finger) is illuminated by the lightcurtain, the object reflects light from the light curtain onto thescreen such that an image of the screen includes a bright spot 605corresponding to the location of the finger with respect to the screen.

Images 610 and 620 are images of the screen, and the images 610 and 620may be collected by, for example, the detection system 430 discussedwith respect to FIG. 4. In the example show, the image 610, whichincludes the bright spot 605 caused by reflection of electromagneticradiation from the finger, is collected at a time t when the finger isilluminated by the light curtain. The image 620, which does not includea bright spot corresponding to the finger, is collected at a time t+1when the finger is not illuminated by the light curtain. For example,the image 610 may be collected while one or more emitters emitelectromagnetic radiation, and the image 620 may be collected while theone or more emitters are extinguishing electromagnetic radiation. Thus,if the finger 605 touches the screen, or is close to the screen suchthat the finger is illuminated by the light curtain, a bright spot 605caused by reflections from finger onto the screen appear in images ofthe screen captured while the finger is illuminated by the flashinglight curtain.

The images 610 and 620 also include bright spots 635 and 640 caused byambient light sources. For example, the bright spots 635 and 640 may bea result of ambient light sources shining directly on the screen orradiation from ambient light sources being reflected onto the screen. Incontrast to the bright spot 605, which is caused by light reflectingfrom a finger and is only present when the finger is illuminated by theflashing light curtain, the bright spots 635 and 640 caused by theambient light sources are included in every captured image. Thus, bysubtracting consecutive frames from each other (such as the images 610and 620) to create a difference image 630, the bright spots 635 and 640caused by ambient light sources are removed, but the bright spot 605caused by reflections from the finger remain.

Because the light curtain illuminates a shallow region in front of thescreen, the tip of the finger is illuminated by the curtain of light,and the tip of the finger reflects light onto the screen. Additionalprocessing beyond subtracting the frames 610 and 620 may be performed todetermine the size, pose, or position of the finger and/or to verifythat the finger is an actual object being illuminated by the flashingcurtain of light. For example, the size and shape of the bright spot 605caused by reflections from the finger may be analyzed to determinewhether a finger is actually present. Additionally, multiple users mayinteract with the application through the light curtain.

Referring to FIG. 7A, a front view of a system 700 is illustrated. Thesystem 700 may be similar to the system 200 discussed above with respectto FIG. 2 and/or the system 300 discussed above with respect to FIG. 3.A screen 705 includes edges 706 a, 706 b, 706 c, and 706 d. In theimplementation shown in FIG. 7, sources 710 a, 710 b, 710 c, and 710 dare positioned at, adjacent to, and/or above the edges 706 a, 706 b, 706c, and 706 d, respectively, and the sources 710 a, 710 b, 710 c, and 710d create a light curtain that substantially covers the screen 705. Thesources 710 a, 710 b, 710 c, and 710 d project the light curtainparallel to the screen 705 and offset from a surface of the screen 705.The sources 710 a, 710 b, 710 c, and 710 d include emitters 711 andchannels 712. In some implementations, the screen 705 may include moreor fewer edges. In some implementations, one or more of the edges of thescreen 705 do not have a corresponding source. For example, in theseimplementations, two edges may have a corresponding source and two edgesdo not have a corresponding source.

Referring to FIGS. 7B-7D, an example emitter 711 that is an LED isshown. In particular, FIG. 7B shows the emitter 711, FIG. 7C shows aside view of the emitter 711, and FIG. 7D shows a front view of an arrayof emitters 711. The emitter 711 emits electromagnetic radiation as acone of light 716. A portion of the cone of light 716 passes through thechannel 712 and emerges as the light curtain. The channel 712 may bemade of a material that blocks light and/or a material that is notreflective. The material that makes the channel 712 may block light ontwo sides of the channel. In some implementations, the channel 712 mayhave sides of glass or plexiglass, either of which may be modified toprevent reflection of light inside the channel. For example, theplexiglass or glass may be ground or sandblasted such that the glass orplexiglass has a diffuse, rough surface. The channel 712 may be filledwith air. In some implementations, the emitters may be line lasers emitradiation that is relatively collimated. In these implementations,because the emitters emit radiation that is not diverging, the lightcurtain may be created without necessarily using the channel 712.

Referring to FIGS. 8A and 8B, an example in which a laser source 810produces a light curtain 815 is illustrated. The laser source 810 may bean infrared laser source. The laser source 810 produces light that doesnot diverge as much as light emitted from an LED. Thus, a channel suchas the channel 712 discussed above may not be necessary to create alight curtain that substantially covers a screen 820. A cylindrical lens825 may be used to make a thin plane of light from the laser 810. Thecylindrical lens 825 distributes a beam of light emitted from the laser810 along a line. The distributed beam makes a thin plane of light 815that is projected parallel to and above a surface of a screen 820, andthe distributed beam effectively covers the screen 820. Referring inparticular to FIG. 8B, an example of an arrangement of line lasers 810with associated cylindrical lenses 825 are shown with the screen 820.The intensity of the light emitted from the line lasers 810 diminishesas the distance from a center point 830 increases. Thus, multiple lasers810 are generally used to effectively cover the screen 820 with thecurtain of light 815 of a particular intensity.

In the example shown in FIG. 8B, portions of the screen are not coveredby the light curtain created from the multiple line lasers. For example,a portion 835 is not covered by the light curtain. Thus, objects infront of the portion 835 of the screen 820 do not reflectelectromagnetic radiation from the flashing light curtain onto thescreen because the flashing light curtain does not illuminate objects infront of the portion 835. As a result, the position of such objects isnot determined. In some implementations, portions of the screen 820 arenot covered by the flashing light curtain intentionally. For example,the portion 835 may correspond to locations in the projectionapplication where the user cannot select an item. The portion 835 maycorrespond to a region in the projection shown on the screen that is notpart of the application.

Referring to FIG. 9, and a system 900 is illustrated. A projector 902projects an image of an application on a screen 905. The application maybe implemented as a software application or hardware module. Flashingelectromagnetic radiation 908 from a source 910 passes through thescreen 905 and illuminates an object 930, and light 912 reflected fromthe object 930 while the object 930 is illuminated by the source 910creates a bright spot 932 on the screen 905 that is imaged by a sensor920. The source 910 is configured to alternate between emitting theelectromagnetic radiation and extinguishing the electromagneticradiation at a rate of flashing. Additionally, an ambient light source935 creates a bright spot 937 that is imaged by the sensor 920. Thesystem 900 tracks a position of the object 930 with respect to thescreen 905 and controls the application based on the position of theobject 930.

Similar to the system 200, the system 900 tracks a position of an object930 with respect to a screen 905 and controls the application based onthe position of the object 930. However, in the system 900, the source910 and the object 930 are located on opposite sides of the screen 905.In particular, the source 910 is located on a first side 940 of thescreen 905, and the object 930 is located on a second side 945 of thescreen 905. Additionally, in this example, the ambient light source 935is located on the second side 945 of the screen 905.

Referring to FIG. 10, a system 1000 determines one or more positions ofan object 1030 as the object 1030 is illuminated by multiple flashinglight curtains 1015, 1016, and 1017. The flashing light curtains 1015,1016, and 1017 are located at distances d1, d2, and d3, respectively,from a surface 1006 of a screen 1005. In the example shown, the object1030 intersects the flashing light curtains 1015, 1016, and 1017 atmultiple locations simultaneously. A snapshot 1020 shows the locationsof the object 1030 with respect to the screen 1005. The multiplepositions may be used to, for example, determine a trajectory of motionof the object 1030. Bright spots 1040, 1041, and 1042 are caused by theobject reflecting light from the flashing light curtains 1015, 1016, and1017, respectively, onto the screen 1005.

Referring to FIG. 11, a system 1100 is illustrated. A projector 1102projects an image of an application 1103 “Grab the Plant” on a screen1105. A source 1110 emits electromagnetic radiation to create a flashingcurtain of light 1115. A sensor 1120 senses electromagnetic radiationreflected by an object 1130 as the object 1130 moves through theflashing curtain of light 1115. In the example shown in FIG. 11, theobject 1130 is a finger of a person, and the object 1130 is shown atthree different times t₁, t₂, and t₃ as the object 1130 moves throughthe light curtain 1115. Electromagnetic radiation from the flashinglight curtain reflects off of the object 1130 onto the screen 1105 asbright spots 1131, 1132, and 1133. An ambient light source (not shown)causes a bright spot 1134 on the screen 1105. Although the bright spot1134 appears on the screen 1105 and is imaged by the sensor 1120, thebright spot 1134 may be removed from the collected data throughtechniques similar to those discussed above. The corresponding positions1161, 1162, and 1163 of a cursor 1140 are also shown at each of thethree times.

The person moves the object 1130 vertically from a position 1151 at timet₁ to a position 1152 at time t₂. The cursor 1140 in the controlledapplication has a corresponding vertical motion from a position 1161 toa position 1162. Between time t₂ and time t₃, the user moves the object1130 vertically again from the position 1152 to a position 1153. Theplant 1170 is projected on the screen 1105 at the position 1153.Additionally, the user also makes a gesture with the object 1130 suchthat the object is pointing upward at time t₃. The gesture made at timet₃ corresponds to a selection (e.g., the equivalent of selecting an itemwith a mouse click) that occurs at a time t₄, which occurs shortly aftertime t₃. Thus, the cursor 1140 in the controlled application has acorresponding vertical motion from the position 1162 to a position 1163.Additionally, the user makes a selection of the plant 1170 through thegesture made at a time t₃. Accordingly, the plant 1170 becomes enlargedat the time t₄ as a result of the user selecting it by making a gesturein the light curtain.

Implementations may include, for example, a process, a device, or adevice for carrying out a process. For example, implementations mayinclude one or more devices configured to perform one or more processes.A device may include, for example, discrete or integrated hardware,firmware, and software. A device may include, for example, computingdevice or another computing or processing device, particularly ifprogrammed to perform one or more described processes or variationsthereof. Such computing or processing devices may include, for example,a processor, an integrated circuit, a programmable logic device, apersonal computer, a personal digital assistant, a game device, a cellphone, a calculator, and a device containing a software application. Thecomputing device may be the computing device 201 discussed above withrespect to FIG. 2.

Implementations also may be embodied in a device that includes one ormore computer readable media having instructions for carrying out one ormore processes. The computer readable media may include, for example,storage device, memory, and formatted electromagnetic waves encoding ortransmitting instructions. The computer readable media also may include,for example, a variety of non-volatile or volatile memory structures,such as, for example, a hard disk, a flash memory, a random accessmemory, a read-only memory, and a compact diskette. Instructions may be,for example, in hardware, firmware, software, and in an electromagneticwave.

Thus, the computing device may represent an implementation of acomputing device programmed to perform a described implementation, andthe storage device may represent a computer readable medium storinginstructions for carrying out a described implementation.

A number of implementations have been described. Nevertheless, it willbe understood that various modifications may be made. Accordingly, otherimplementations are within the scope of the following claims.

What is claimed is:
 1. A computer-implemented method comprising:accessing first and second images of an object, the first image beingcaptured while the object is illuminated with projected electromagneticradiation, and the second image being captured while the projectedelectromagnetic radiation is extinguished; projecting at least a portionof the projected electromagnetic radiation as a first light curtainalong and parallel to at least a portion of a first side of a screen;determining a position of the object relative to the screen based oncomparing the first and second images; and controlling an applicationbased on the determined position.
 2. The method of claim 1, furthercomprising: projecting electromagnetic radiation as a second lightcurtain adjacent and parallel to the first side of the screen, the firstlight curtain and the second light curtain being parallel to each otherand disposed at a first location and a second location with respect tothe first side of the screen.
 3. The method of claim 1, wherein: atleast a portion of the projected electromagnetic radiation reflects offof the object and illuminates a portion of the screen; and determining aposition of the object includes identifying the illuminated portion ofthe screen.
 4. The method of claim 3, wherein the determining theposition of the object includes identifying a size and shape of theilluminated portion of the screen.
 5. The method of claim 1, furthercomprising determining a second position, and wherein the application iscontrolled based on the determined position and the second position. 6.The method of claim 1, wherein: the first image and the second image arecaptured on one side of the screen; and the object is disposed on asecond side of the screen.
 7. The method of claim 1, further comprisingprojecting electromagnetic radiation with at least one of alight-emitting diode (LED) or a laser.
 8. The method of claim 1, whereinthe projected electromagnetic radiation comprises infrared radiation. 9.The method of claim 1, wherein a visible image is displayed on thescreen, the object is in motion, and controlling an application based onthe determined position comprises controlling a cursor on the visibleimage such that motion of the cursor corresponds to the motion of theobject.
 10. The method of claim 9 wherein the motion of the objectcomprises a gesture made by a person.
 11. The method of claim 1, whereinaccessing first and second images comprises accessing first and secondimages of an object and a second object, the object and the secondobject disposed on the same side of the screen, and further comprisingdetermining a position of the second object relative to the screen. 12.The method of claim 1, further comprising modeling a shadow cast on thescreen, the shadow cast by the object while the object is illuminatedwith the projected electromagnetic radiation.
 13. The method of claim12, further comprising determining, based on the model, that the shadowobscures a second object and further comprising determining a positionof the second object relative to the screen based at least in part onaccounting for the shadow.
 14. A system comprising: a screen; a sourceof electromagnetic radiation, the source configured to: alternatebetween emitting the electromagnetic radiation and extinguishing theelectromagnetic radiation at a rate of flashing, and project theelectromagnetic radiation as a light curtain along and parallel to asurface of the screen; a sensor configured to sense electromagneticradiation at the rate of flashing when the electromagnetic radiation isreflected from an object disposed in the vicinity of the screen; and aprocessor configured to: determine a position of the object based on thesensed electromagnetic radiation, and control an application based onthe determined position of the object.
 15. The system of claim 14,wherein the screen comprises edges, and the source of electromagneticradiation comprises at least one emitter positioned along one or more ofthe edges of the screen.
 16. The system of claim 15, wherein the sourceof electromagnetic radiation comprises one or more arrays of emitterspositioned along one or more of the edges of the screen.
 17. The systemof claim 14, wherein the sensor comprises an array of pixels.
 18. Thesystem of claim 14, wherein the source of electromagnetic radiationcomprises a series of linear arrays of emitters, each of the lineararrays positioned on a certain side of the screen at a distance from thesurface of the screen.
 19. The system of claim 14, wherein the source ofelectromagnetic energy emits diverging electromagnetic energy, andfurther comprising a channel adjacent to the source, the channelconfigured to collimate the emitted electromagnetic energy.
 20. Thesystem of claim 19, wherein the channel comprises a block of materialwith a passageway extending through the block from an entry to an exit,the diverging electromagnetic energy impinging on the entry and aportion of the diverging electromagnetic energy emerging from the exitas collimated electromagnetic energy.
 21. The system of claim 19,wherein the channel comprises a lens.
 22. The system of claim 14,wherein the source of electromagnetic radiation comprises an emitterpositioned on a first side of the screen, the source of electromagneticradiation configured to emit electromagnetic radiation that passesthrough the screen and interacts with the object disposed on a secondside of the screen.
 23. The system of claim 14, wherein the source ofelectromagnetic radiation comprises a light emitting diode.
 24. Thesystem of claim 23, wherein the light emitting diode emits infraredradiation.
 25. The system of claim 14, wherein the processor is furtherconfigured to model a shadow cast by the object and determine a positionof a second object disposed on the second side of the screen based onthe modeled shadow.
 26. A computer program product, embodied in anon-transitory machine-readable medium, the computer program productcomprising instructions that, when read by a machine, operate to cause adata processing apparatus to: project electromagnetic radiation as afirst light curtain along and parallel to at least a portion of a firstside of a screen; access first and second images of an object, the firstimage being captured while the object is illuminated with the projectedelectromagnetic radiation, and the second image being captured while theprojected electromagnetic radiation is extinguished; determine aposition of the object relative to the screen based on comparing thefirst and second images; and control an application based on thedetermined position.
 27. The computer program product of claim 26,further including instructions that operate to cause the data processingapparatus to determine a second position, wherein the application iscontrolled based on the determined position and the second position. 28.The method of claim 26, further comprising instructions that operate tocause the data processing apparatus to model a shadow cast on thescreen, the shadow cast by the object while the object is illuminatedwith the projected electromagnetic radiation.
 29. A system comprising:means for projecting electromagnetic radiation as a light curtain in thevicinity of a screen, wherein the means for projecting electromagneticradiation is configured to alternate between emitting theelectromagnetic radiation and extinguishing the electromagneticradiation at a rate of flashing; means for sensing electromagneticradiation at the rate of flashing when the electromagnetic radiation isreflected from an object disposed in the vicinity of the screen; meansfor determining a position of the object based on the sensedelectromagnetic radiation; and means for controlling an applicationbased on the determined position of the object.
 30. The system of claim29, wherein the means for projecting electromagnetic radiation emitsdiverging electromagnetic energy, and further comprising means forcollimating the emitted electromagnetic energy.
 31. The system of claim29, further comprising means for: modeling a shadow cast by the object,and determining a position of a second object based on the modeledshadow.